Did some testing using this branch between a gaia bin on v4.2.1 and the hub/cosmoshub-4.6 branch. Everything works great! I do get some "packet already received" errors when running hermes start-multi. See logs. Is this expected?

The main downside is lost fees for retrying these transactions

Edit: It works fine if I use gaiad to initiate the transfer. Must be related to hermes tx raw

Did some testing using this branch between a gaia bin on v4.2.1 and the hub/cosmoshub-4.6 branch. Everything works great! I do get some "packet already received" errors when running hermes start-multi. See logs. Is this expected?

The main downside is lost fees for retrying these transactions

Edit: It works fine if I use gaiad to initiate the transfer. Must be related to hermes tx raw

This is not expected unless you had another relayer running in parallel. Could you give more detail of the setup? And the exact ft-transfer command you used? I use this command often and haven't seen this before.

I setup an old/new gaia using the gaiad manager. Created channels and sent tokens using ts-relayer. Then stopped the ts-relayer (I'm fairly certain)

Created channels using hermes, started hermes via start-multi. Sent using

hermes tx raw ft-transfer network2 network1 transfer channel-0 1 -o 1000 -n 1 -d localcoin

got the above logs.

Then sent using gaiad afterwards and it worked fine.

I will try to reproduce again later today

I realize that the transfer is being sent over channel-0. I created channel-0 with ts-relayer and channel-1 with hermes

I managed to reproduce. The problem scenario is when the ts-relayer is also running. Specifically:

• start ts-relayer
• send packet
• wait for ts-relayer to log that it has relayed the packet
• start hermes

Then hermes will actually beat the ts-relayer to the acknowledgement process but get stuck on trying to relay the packet

Does hermes rely on the ibc-go error codes at all? We are planning on making the error numbers as apart of our API for versioning so we don't change them often and when we do it is well documented. We can turn the unordered channel packet already received into an error code so that hermes could stop retries. Would this help?

Does hermes rely on the ibc-go error codes at all? We are planning on making the error numbers as apart of our API for versioning so we don't change them often and when we do it is well documented. We can turn the unordered channel packet already received into an error code so that hermes could stop retries. Would this help?

Thanks @colin-axner ! We are working on better error filtering and definitely having the error code here will help.

BTW, this also happens with two hermes relayers running. We haven't really started doing these types of tests heavily and understand the issues. But it's coming.

opened a pr to update the error messages, also opened an issue on the spec repo

Please checkout the ibc-go pr and see if there is anything missing that'd be useful. The only scenario I could think of not covered is already timed out packets on ordered channels. This is tricky since by construction only 1 packet can be timed out at the moment (since the channel closes). Since ICS20 doesn't use ordered channels, I'd prefer to wait to fix until more than 1 packet can be timed out

Opened #984 to check if the event has been previously handled by another relayer. This should reduce the number of retries to one. I tested with two hermes instances and the ft-transfer command.

Some context on this PR: Even if we build Hermes with these new proto files (SDK 0.43 and ibc-go v1), the relayer should still be able to submit chain upgrade proposals and upgrade clients with 0.42-based chains (gaiad 5.0.5) for instance.

So merging this PR should not break any CLI and we should retain backwards compat with 0.42-based chains. We need to test for that.

Update: Romain's tests reminded me that my statement above was wrong. Issue #1209 documents that we'll need an explicit --legacy flag to maintain backwards compatibility when it comes to the chain upgrade CLI.

Another thing is that we should update compatibility.rs to mark 0.43 as a compat. version.

Running the following command using Hermes built from this branch and two v4.2.1 Gaia nodes setup via gm yields:

❯ hermes -c ~/.hermes/gm-config.toml tx raw upgrade-chain network1 network2 07-tendermint-0 10000000 3000

Aug 12 14:53:15.911  INFO Message UpgradePlanOptions { src_chain_config: ChainConfig { id: ChainId { id: "network2", version: 0 }, rpc_addr: Url { inner: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(27030), path: "/", query: None, fragment: None }, scheme: Http, host: "localhost", port: 27030 }, websocket_addr: Url { inner: Url { scheme: "ws", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(27030), path: "/websocket", query: None, fragment: None }, scheme: WebSocket, host: "localhost", port: 27030 }, grpc_addr: Url { inner: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(27032), path: "/", query: None, fragment: None }, scheme: Http, host: "localhost", port: 27032 }, rpc_timeout: 10s, account_prefix: "cosmos", key_name: "wallet", store_prefix: "ibc", max_gas: Some(500000), gas_adjustment: None, max_msg_num: MaxMsgNum(30), max_tx_size: MaxTxSize(2097152), clock_drift: 5s, trusting_period: 1209600s, trust_threshold: TrustThresholdFraction { numerator: 1, denominator: 3 }, gas_price: GasPrice { price: 0.001, denom: "stake" }, packet_filter: AllowAll }, dst_chain_config: ChainConfig { id: ChainId { id: "network1", version: 0 }, rpc_addr: Url { inner: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(27020), path: "/", query: None, fragment: None }, scheme: Http, host: "localhost", port: 27020 }, websocket_addr: Url { inner: Url { scheme: "ws", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(27020), path: "/websocket", query: None, fragment: None }, scheme: WebSocket, host: "localhost", port: 27020 }, grpc_addr: Url { inner: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(27022), path: "/", query: None, fragment: None }, scheme: Http, host: "localhost", port: 27022 }, rpc_timeout: 10s, account_prefix: "cosmos", key_name: "wallet", store_prefix: "ibc", max_gas: Some(500000), gas_adjustment: None, max_msg_num: MaxMsgNum(30), max_tx_size: MaxTxSize(2097152), clock_drift: 5s, trusting_period: 1209600s, trust_threshold: TrustThresholdFraction { numerator: 1, denominator: 3 }, gas_price: GasPrice { price: 0.001, denom: "stake" }, packet_filter: AllowAll }, src_client_id: ClientId("07-tendermint-0"), amount: 10000000, height_offset: 3000, upgraded_chain_id: ChainId { id: "network1", version: 0 }, upgraded_unbonding_period: None, upgrade_plan_name: "plan" }

Error:
   0: upgrade chain error
   1: tx response event consists of an error: deliver_tx on chain network1 for Tx hash 6FCF5B01B97296451F774BCD2DE8CEE47CA16E6F84CC3A6A5AE4F33249A1B05B reports error: code=Err(2), log=Log("unable to resolve type URL /ibc.core.client.v1.UpgradeProposal: tx parse error")

The issue is likely that

• SDK 0.42 expects a /cosmos.upgrade.v1beta1.SoftwareUpgradeProposal message
• SDK 0.43 expects a /ibc.core.client.v1.UpgradeProposal message

UPDATE: This is already tracked in #1209 and will be addressed in a follow-up PR.

I have tested this branch against all gaia versions v4.0.3 onwards, i.e. SDK version >=0.41.3. All E2E tests are passing.

Also ran the E2E tests on some heterogeneous setups ->

• v4.0.3 <> colin-fee-event-testing (i.e. earliest v/s latest SDK versions supported)
• v5.0.5 <> colin-fee-event-testing (latest release v/s latest supported)
• v5.0.5 <> v4.0.3 (latest release v/s earliest supported)
• v4.1.2 <> colin-fee-event-testing (possibly widely deployed)
• v5.0.5 <> v5.0.2 (random)

All tests are passing. ✅